Skip to content

Conversation

@tanmaya-panda1
Copy link
Collaborator

This pull request introduces significant changes to the ingestion client and uploader logic to simplify and clarify multi-source ingestion, enforce stricter type safety, and improve concurrency handling. The main focus is to restrict multi-source ingestion to only support blob sources, remove internal logic for uploading local sources within multi-ingest, and enhance the uploader's parallelism and load distribution.

Multi-source ingestion API and logic changes:

  • Restricts multi-source ingestion (ingestAsync) to only accept BlobSource objects, both in the interface and implementation, and updates all related documentation to clarify that local sources must be uploaded separately before ingestion. This prevents partial failure scenarios and enforces a clearer, safer workflow. [1] [2] [3] [4]
  • Removes the internal method for uploading local sources (uploadLocalSourcesAsync) from QueuedIngestClient, making it the caller's responsibility to upload local files/streams before calling multi-ingest. [1] [2]

Error handling and messaging improvements:

  • Updates error messages in IngestClientException to use the message parameter for clarity and consistency. [1] [2]

Uploader concurrency and load balancing enhancements:

  • Switches the container selection logic in ContainerUploaderBase from random to round-robin using an atomic counter, ensuring more even load distribution across containers during uploads and retries. [1] [2] [3]
  • Improves parallel upload logic to use a semaphore for true streaming parallelism, allowing new uploads to start as soon as slots are available, rather than processing in fixed-size chunks.

Java interop and API usability:

  • Adds overloaded static factory methods to DefaultConfigurationCache for easier creation from Java, supporting both minimal and full-parameter configurations.
  • Adds @JvmOverloads constructors to FileSource and StreamSource for better Java interoperability and sensible defaults. [1] [2]

These changes collectively make the ingestion API safer, simpler, and more robust, especially in multi-source scenarios.### Added

Changed

Fixed

@github-actions
Copy link

github-actions bot commented Jan 15, 2026

Test Results

530 tests  ±0   521 ✅ ±0   9m 0s ⏱️ +50s
 31 suites ±0     9 💤 ±0 
 31 files   ±0     0 ❌ ±0 

Results for commit 9be5b23. ± Comparison against base commit ce36f64.

♻️ This comment has been updated with latest results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants